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Examples to be covered 

1. "SURVEY" Inspection to Detect Damage 

- Rapid, preplanned coverage of large areas 

- Detect possible holes, cracks for focused inspection 

2. "FOCUSED INSPECTION" to Quantify Damage 

- Extended-time coverage of detections 

- Measure damage depth/volume, especially at critical 
subsurface layers 

3. "VIBROMETRY" 

- Remotely measure modal response of structure 

• Validate structural models 

• Detect damage-induced changes 


Application 1: SURVEY 

Closed Circuit TV video was used for Shuttle Survey 

- Rapid area coverage, real-time downlink, many remote 
camera stations (including SSRMS) 

Desire for higher resolution is constrained by 
existing data transmission hardware 

- Incentive to enhance imagery via software, if possible 

Examples in following slides 

- Shuttle Leading Edge Inspection, STS114-STS135 

• LDRI video (an intensified camera) 

- International Space Station video 

• Color TV video 

• VIPIR video 


Image Enhancement Examples 


• Pixel Gain and Offset Correction 

• Camera distortion correction 

• Motion stabilization and multiframe averaging 

• Inverse Blur correction 

• Super resolution 



PIXEL GAIN AND OFFSET CORRECTION 


Applied to LDRI CAMERA 
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DISTORTION CORRECTION 


• Distortion causes straight lines to appear curved 

• Desired for photogram metry or highly distorted images 

• Corrected using calibrated distortion parameters 

— Brown's Distortion Model 
- Laboratory calibration or data-driven 


Multi-frame Averaging + Contrast Stretching 

(LDRI Camera) 

• Normal pixel intensity is 8-bits, with noise 

• Averaging improves intensity resolution with 
fractional resolution and reduced noise 



• On orbit inspection of OV-103 Port Panel 10 (RCC) 




Multi-frame Averaging + Filtering 

(LDRI Camera) 


• Reduced noise/aliasing improves spatial resolution 

• Enables inverse filtering to improve blur 
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Multi-frame Averaging + Filtering 

(LDRI Camera) 

P6 Truss of the International Space Station, STS97 



Multi-frame Averaging + Filtering 

(LDRI Camera) 


1951 USAF resolution up by ~26% (2 elements) 



POTENTIAL OF SUPER RESOLUTION 

(simulated 30x30 dat a) 
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SUPER RESOLUTION OF ISS VIDEO 






POTENTIAL OF CHANGE DETECTION 
(Background Subtraction) 




• Sensitive to nearly invisible features 

- Light scuff marks and sub-pixel objects 

- Good results require close positioning, and 
minimizing lighting changes 


"Blink" Change Detection 

STS 130 Inspection, LDRI Camera 



FD2 


Late 




Change Detection 


STS135 Inspection, LDRI Camera 



This slide is unfinished, but will show 
the difference in the above images 



Data-Driven Background Subtraction 


• Estimate background and subtract from image 
- Estimate background as narrow-band spectral peaks 


BACKGROUND 
RAW IMAGE ESTIMATION 



BACKGROUND 

SUBTRACTION 



Tian, Xuwen: "Data-driven textile flaw 
detection methods" 



Change Detection Applied to RADAR 

• Radars can measure complex reflectance from scene "pixels" 

• Radar images (left, center) show reflectance magnitude 

• CCD image (right) shows change in complex reflectance 

- Can be much more sensitive to change than magnitude 

• Might this enable radar to be used for SURVEY? 

- Could sub-pixel, sub-wavelength damage be detected? 



Pre-activity image Post-activity image CCD image 


From "C4ISR for Future Naval Strike Groups", NRC, 2006 


Application 2: FOCUSED INSPECTION 
Measure Damage Size and Depth 

Camera Suitability 

• Incentive to use Cameras 

- Existing resource, robotic flexibility 
— High resolution, visual context 

• Issues with Cameras 

- Depth measurement required 

- Non-penetrating, limited to line-of-sight 

— Illumination difficulty into narrow openings 
-Tradeoff between Resolution and Depth of focus 


RESOLUTION VS. DEPTH OF FOCUS 

(500nm Wavelength) 



LENS DIAMETER / AXIAL STANDOFF 


^—Diffraction Limit 

0.05mm DOF 

—0.10mm DOF 

0.20mm DOF 

0.50mm DOF 
— 1.00mm DOF 
2.00mm DOF 

5.00mm DOF 

10.0mm DOF 

20.0mm DOF 

50.0mm DOF 


Examples Yielding 
lOOum Resolution, 20mm DOF 

(FOCAL LENGTH AS REQUIRED) 

50mm (2") Standoff: 0.25-0. 50mm Lens Diameter 

- "Borescope" type of camera 

- Requires special prox ops approval 

500mm (20") Standoff: 2. 5-5. 0mm Lens Diameter 

- Nikon or video camera at high F/#, small telephoto lens 

- Dextre prox ops 

lm (3.1') Standoff: 5.0-10mm Lens Diameter 

- SSRMS prox ops 

10m (16') Standoff: 50-100mm Lens Diameter 

- Nikon or video camera at low F/#, large telephoto lens 




Hypervelocity Impacts to Sample Silica Re-entry Tile (#3) 

Entry Holes and X-ray CT Images 


(Courtesy NASA/KX) 


22 






COLORIMETRY (Nikon Camera) 


Color in a critical layer can be a sensitive damage indication 
In this case, the critical layer is colored red 




FLUORESCENCE 


The critical damage site may not always be visible 
Fluorescent material from a critical layer can be a sensitive 
indicator in the visible debris 



(Source: Michael Rollins, JSC Image Sciences) 


3D Evaluation 


Goal: Measure size and depth of ID1, ID2, ID3 

Data Base: 2-D camera images, multiple poses 

- Borescope Camera (~1" standoff) 

- Nikon Camera (~20" standoff) 

Depth determination 

- Using parallax: "Structure From Motion" 

- Using focus: "Structure from Focus" 



Borescope Testing 



Borescop^ Configuration. Pulnix camera (at right) is optically coupled to distal end of 8-mm 
borescope (at left). Illumination is coupled through flexible light guide (at center). 



— Diffraction Limit 
0.05mm DOF 

0.10mm DOF 

0.20mm DOF 

0.50mm DOF 

1.00mm DOF 

2.00mm DOF 

5.00mm DOF 

10.0mm DOF 

20.0mm DOF 

50.0mm DOF 


Illumination ring 
surrounding input window 


o = 20-40mm range, 1mm aperture, 60° FOV 


ID1 Structure From Motion 


• 20mm normal standoff 

• Center, 4mm up, down, left, right 



TRUTH DATA 

(Studor/Olstad/Smith/Rollins: “MMOD Inspection and 
Detection Study and Recommendations” ) 


BORESCOPE SFM DATA 
(PhotoScan software analysis ) 




ID1 Structure From Motion 


• 20mm normal standoff 

• Center, 4mm up, down, left, right 



TRUTH DATA 

(Courtesy Michael Rollins, JSC Image Sciences) 


BORESCOPE SFM DATA 
(PhotoScan software analysis ) 




ID2 Structure From Motion 


17-23mm axial standoff 
Conical scans at 30,40,50° 
10° azimuth steps 


I'D IB 





TRUTH DATA 

(Studor/Olstad/Smith/Rollins: “MMOD Inspection 
and Detection Study and Recommendations” ) 


BORESCOPE DATA RECONSTRUCTION 
(PhotoScan software analysis ) 






ID2 Structure From Motion 



TRUTH DATA 

(Courtesy Michael Rollins, JSC Image Sciences) 


BORESCOPE DATA RECONSTRUCTION 
(PhotoScan software analysis ) 




ID3 Structure From Motion 


• 8-10mm axial standoff 

• Conical scans at 30, 40° 

• 10° azimuth steps 


TRUTH DATA 

(Studor/Olstad/Smith/Rollins: “MMOD Inspection 
and Detection Study and Recommendations” ) 



BORESCOPE DATA 
(PhotoScan software analysis ) 





ID3 Structure From Motion 


8-10mm axial standoff 
Conical scans at 30, 40° 
10° azimuth steps 



TRUTH DATA BORESCOPE DATA 

(Courtesy Michael Rollins, JSC Image Sciences) (PhotoScan software analysis ) 






Optical Resolution (um in Object Plane) 


Nikon Testing 



Nikon DX2s camera with Micro- Nikkor 105mm f/2.8 lens and ring illuminator (at right) viewing 
AETB-8 target mounted on computer-controlled, translation and rotation stage (left). 


Optical Resolution 

500nm Wavelength 


1000 


100 


10 



0.01 


0.1 


Diffraction Limit 

0.05mm DOF 

0.10mm DOF 

0.20mm DOF 

0.50mm DOF 

1.00mm DOF 

2.00mm DOF 

5.00mm DOF 

10.0mm DOF 

20.0mm DOF 

50.0mm DOF 


o 

o 


= 300-500mm range, 37 mm 
aperture, 5.5um pixels 

= 500mm range, 3 mm aperture, 
5.5um pixels (4-pixel blur) 


1 — 
0.001 


Numerical Aperture 


1 


Borescope vs Nikon 
Comparison 


BORESCOPE FROM 10mm (0.4") 


NIKON FROM 0.44m (17.3") 


ID3 Structure From Motion 


(Nikon Dataset Limited to just 5 views) 


• 266mm (17.3") axial standoff 

• 45° incidence 

• 5 images at 20mm steps 



TRUTH DATA 

(Studor/Olstad/Smith/Rollins: “MMOD Inspection 
and Detection Study and Recommendations” ) 



NIKON DATA 

(PhotoScan software analysis ) 



ID3 Structure From Motion 


(Nikon Dataset Limited to just 5 views) 


• 266mm (17.3") axial standoff 

• 45° incidence 

• 5 images at 20mm steps 



TRUTH DATA 

(Studor/Olstad/Smith/Rollins: “MMOD Inspection 
and Detection Study and Recommendations” ) 



NIKON DATA 

(PhotoScan software analysis ) 



STRUCTURE FROM FOCUS 



3D from 2D Conclusions 


Proper illumination and lens are crucial 

Over-the-counter Structure From Motion software performs well 

- Further validation needed 

• More examples, difficult cases 

• Scaling and dimensional comparisons with truth 

- Structure from Focus also appears feasible 

• Was investigated in less detail 

• May complement SFM 

Potential improvement by combining focus, motion algorithms 

- Results presented here use either but not both simultaneously 

Best results require many views 

- Easily configured using robotics 

- Could be performed with either video or Nikon 

- Processing in near-real time 

• ~10min on laptop 

• Estimated 30s on high performance desktop 



RADAR Observations 

• Lower frequencies penetrate better 

• Return from reflective backing preferred to 
direct return 

• Circular polarization preferred to linear/none 

• Polarization ratio preferred to magnitude of 
either polarized or unpolarized return 


"Insitu Damage Imaging of Inflatable Structures" 
Studor, Madaras, Nellums, McMakin 


10-20 GHz Radar, Circularly Polarized 

(Courtesy Douglas McMakin, PNNL) 


Direct Return 
from Undamaged 
Surface 


Reflected 
Return from 
MLI Backing 



R-R R-L/R-R 



DISPLACEMENT (PIXELS) 


APPLICATION 3: VIBROMETRY 


Identify modes to validate codes and detect changes 



Approach: 

Adjust each video frame, using a 
motion model, to best match a 
reference image. 

Key Need: 

Adequately identify more modes 


DX,DY Displacement, Blanket Box of S43A-SAW, Frames 0-16828 
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0 2000 4000 6000 8000 10000 12000 14000 16000 18000 

FRAME NUMBER 

("Response of ISS S4-3A Solar Array Wing to Reboost 6/17/2011" , courtesy Donn Liddle, NASA/JSC Image Sciences) 


VIBROMETRY CONCLUSIONS 
Potential Areas for Improvement 

Registration Algorithm 

- Especially regarding motion interpolation 

Video Error Correction 

- Many systematic issues with downlinked video 

Motion Model Degrees of Freedom 

- Current model is offset only 

Reference Image Accuracy and Resolution 

- Effect of aliasing and desire for super-resolution 



Spectral Comparison of Results 


Spectrum of 
Existing Analysis 

(floor =0.01 pixels/root-Hz) 



Spectrum of 
Refined Analysis 

(floor =.0024 pixels/root-Hz) 



Frequency (Hz) 


END 



